Method and apparatus for a compact and high resolution mind-view communicator

ABSTRACT

An embodiment of a mind-view communication apparatus includes a first portable unit and a second portable unit. The first portable unit includes an eyeglass frame, at least one first optical unit disposed on the eyeglass frame for capturing at least one scene image corresponding to a field of view of a user, and at least one second optical unit disposed on the eyeglass frame for capturing at least one eye image corresponding to at least a portion of at least one eye of the user. The second portable unit is in communication with the first portable unit and includes at least one processor configured for receiving the at least one scene image and the at least one eye image, determining a direction within the field of view to which the at least one eye is directed based upon the at least one eye image, and generating a subset of the at least one scene image based on the determined direction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/400,399, filed Jan. 6, 2017, and entitled METHOD AND APPARATUS FOR ACOMPACT AND HIGH RESOLUTION MIND-VIEW COMMUNICATOR, now U.S. patent Ser.No. 10/064,552 to be issued on Sep. 4, 2014, which is a continuation ofU.S. patent application Ser. No. 13/175,421, filed Jul. 1, 2011, andentitled METHOD AND APPARATUS FOR A COMPACT AND HIGH RESOLUTIONMIND-VIEW COMMUNICATOR, which is a continuation-in-part of U.S. patentapplication Ser. No. 12/794,283, filed Jun. 4, 2010, and entitled METHODAND APPARATUS FOR A COMPACT AND HIGH RESOLUTION EYE-VIEW RECORDER, nowU.S. Pat. No. 8,872,910 issued on Oct. 28, 2014, which claims thebenefit of U.S. Provisional Application No. 61/184,232, filed Jun. 4,2009, and entitled METHODS AND APPARATUS FOR A COMPACT AND HIGHRESOLUTION EYE-VIEW RECORDER, all of which are incorporated by referenceherein. U.S. patent application Ser. No. 13/175,421 claims the benefitsof U.S. Provisional Application No. 61/369,618, filed Jul. 30, 2010, andentitled “Applications for a Compact and High Resolution Eye-ViewRecorder”; U.S. Provisional Application No. 61/471,397, filed Apr. 4,2011, and entitled “Eye Tracking Device, Apparatus, and Algorithms”; andU.S. Provisional Application No. 61/369,618, filed Apr. 4, 2011, andentitled “Software-Enabled High Resolution Compact Video Recorder” allof which are incorporated by reference herein.

TECHNICAL FIELD

Embodiments of the invention relate to devices and methods for recordingand broadcasting what a user senses. The minimum monitored senses arehearing and vision and the resultant device for these two parameters isa personal video recorder. Pair of eyeglasses frame with integratedopto-electronic devices and a pocket size control unit records the worldlike a human mind. In essence, this device records short term visual andaudio memory in an external device. Autonomous recording of short termmemory can be utilized as a medical device for people who suffer fromshort term memory loss to complement their brain. Live broadcast of suchvideo can be utilized by caregivers of memory patients or by peers on asocial network or by real-time collaborators.

BACKGROUND

Cameras and camcorders are two main devices that people use to takepictures and create movies. To use these devices, one uses a viewfinderor display to select a scene or frame. As one is engaged in sceneselection, he/she concentrates on what is being recorded. This is finefor professionals whose main job is taking photos or recording movies.However, the majority of camera and camcorder users are individuals whouse these devices for personal use. For example, parents videotape theirchildren during birthday parties and other special occasions such aschildren's performances at schools. As one tries to capture a momentcarefully, he/she has to split his attention between recording the eventand enjoying the experience. In effect, there is a contradiction betweenfocusing on recording and enjoying the experience fully. Additionally,existing image and video recorder devices cannot be carried around allthe time because of their bulk and weight; consequently, many unexpectedand one-of-a-kind moments are not recorded. Hence, there is a need foran autonomous device that can record what a user sees, senses andexperiences.

SUMMARY

An embodiment of a mind-view communication apparatus includes a firstportable unit and a second portable unit. The first portable unitincludes an eyeglass frame, at least one first optical unit disposed onthe eyeglass frame for capturing at least one scene image correspondingto a subset of the user's total field of view, and at least one secondoptical unit disposed on the eyeglass frame for capturing at least oneeye image corresponding to at least a portion of at least one eye of theuser. The second portable unit is in communication with the firstportable unit and includes at least one processor configured forreceiving the at least one scene image and the at least one eye image,determining a direction within the field of view to which the at leastone eye is directed based upon the at least one eye image, andgenerating a subset of the at least one scene image based on thedetermined direction.

An embodiment of a method for capturing and processing images includesthe steps of capturing at least one scene image corresponding to a fieldof view of a user by at least one first optical unit disposed on aneyeglass frame, capturing at least one eye image corresponding to atleast a portion of at least one eye of the user by at least one secondoptical unit disposed on the eyeglass frame, receiving the at least onescene image and the at least one eye image by at least one processor,determining a direction within the field of view to which the at leastone eye is directed based upon the at least one eye image, andgenerating a subset of the at least one scene image based on thedetermined direction.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding, reference is now made to thefollowing description taken in conjunction with the accompanyingDrawings in which:

FIG. 1 illustrates simplified block diagrams of an embodiments of themind-view recorder (MVC);

FIG. 2 illustrates simplified block diagrams of another embodiment ofthe mind-view recorder (MVC);

FIG. 3a illustrates a diagram of an embodiment of an MVC frame;

FIGS. 3b and 3c illustrate embodiments of cameras and infra-red LEDconfigurations for the MVC;

FIG. 4a illustrates a schematic diagram of an embodiment in which cameraoutputs are transmitted from the eyeglasses frame to the electronic boxvia a fiber;

FIG. 4b illustrates the use of tapered fused imaging fibers to couplelights from more than one imaging lens to a single imaging sensor;

FIG. 5 illustrates an eye tracking algorithm;

FIGS. 6a-6c illustrate steps of acquiring an image, finding an optimumthreshold via histogram and a detected pupil;

FIG. 7 illustrates an embodiment of a simplified and CPU-efficientalgorithm for pupil detection which can be implemented in hardware;

FIG. 8 illustrates a numerical example of a second algorithm for pupildetection which can be programmed into a camera on a chip module;

FIG. 9 illustrates an image of an eye that is illuminated with multiIR-LED;

FIG. 10 shows the calculated Row Vector for the image shown in FIG. 9;

FIG. 11 shows the calculated Column Vector for the image shown in FIG.9;

FIG. 12 shows that calculated derivative of the Row Vector;

FIG. 13 shows that calculated derivative of the Column Vector;

FIG. 14 shows the calculated location of the center of the pupil on theimage;

FIG. 15 illustrates an embodiment of a simplified and CPU-efficientalgorithm for pupil detection which can be implemented in hardware;

FIG. 16 shows the relative field of views captured by the two wide andnarrow angle cameras in which the narrow angle camera captures a highresolution image (q2) of a subset of the wide angle view camera (g);

FIGS. 17a and 17b show two methods to achieve higher resolution samplingof the field of view through orientation diversity (17 a) and viaorthogonal orientation of rectangular pixels (17 b);

FIG. 18 illustrates a scene of interest with respect to the field ofview of two cameras in which image data from the high resolution area Nwill be used to increase the resolution of the area denoted by A;

FIGS. 19a-19c illustrate alternative configurations for high resolutionsampling of whole field of view via more than one camera wherein FIGS.19a, 19b and 19c five, three and two cameras, respectively, are used tosample the FOV of interest;

FIGS. 20a-20c illustrate an embodiment of a four-camera high resolutioncamera configuration to cover the whole field of view directly in whichin FIGS. 20a and 20b the left and right camera modules FOV and theiroverlap are shown, and in FIG. 20c the net FOV with redundancies areshown;

FIG. 20d illustrates an embodiment of a two high resolution camerasolution to cover the entire field of view of interest;

FIGS. 21a-21d illustrate graphically the concept of super resolutionimage enhancement via sub-pixel image shift across the image sensor;

FIGS. 22a-22c depict embodiments of two imaging configurations to shiftan image across an image sensor in which FIG. 22a shows a standardconfiguration for a lens and an imaging device, and FIG. 22b and FIG.22c illustrate the key additional hardware to achieve image steering;

FIG. 23 illustrates lateral image displacement ranges that can beachieved with a single glass slab that can be positioned at variousangles with respect to the imager;

FIG. 24a illustrates image pixels in a standard camera; and

FIG. 24b shows the virtual pixels that are obtained by diagonaldisplacement of the image across the imager and their locations withrespect to the real pixels.

DETAILED DESCRIPTION

Referring now to the drawings, wherein like reference numbers are usedherein to designate like elements throughout, the various views andembodiments of METHOD AND APPARATUS FOR A COMPACT AND HIGH RESOLUTIONMIND-VIEW COMMUNICATOR are illustrated and described, and other possibleembodiments are described. The figures are not necessarily drawn toscale, and in some instances the drawings have been exaggerated and/orsimplified in places for illustrative purposes only. One of ordinaryskill in the art will appreciate the many possible applications andvariations based on the following examples of possible embodiments.

To address the issue with current cameras and camcorders, embodiments ofa wearable and fully automatic video recorder are described herein. Asthe name “Mind-View Communicator” (MVC) may imply, embodiments of thisvideo recording device view the world like a human eye. It has a similarField Of View (FOV) and zooming capabilities as those of human eyes.

Various embodiments split the video recording device into two parts: anoptical unit that views the world and an electronic box that containsprocessors, storage, battery, LCD display, user interfaces andcommunication ports. The optical portion is fitted within an eyeglassesframe and thus becomes wearable. The electronic box can, for example,fit in a pocket or can be worn like a necklace. The box communicateswith the frame through either a wired or wireless connection. In a caseof wired connection, the user may carry the electronic box, for example,in a pocket. When the box and the frame are connected wirelessly, invarious embodiments the eyeglasses frame may be all that is worn by theuser.

A feature of various embodiments of the MVC is recording precisely whatits user is viewing. To do this, MVC uses eye tracking to follow theuser's eyes for scene selection. In this way, the camera records onlythe frame that the user is looking at. The user can initiate therecording manually by pushing a button on the electronic box or canchoose an attention monitoring circuitry to trigger the recording. Inthe later case, the recording will start automatically as soon assomething that interests the user is detected.

In at least one embodiment, the MVC has four distinct building blocks:the Composite Eye (CE), the Eye Tracking (ET), the Sensing Unit (SU) andthe Electronic Box (EB). The CE views the world and captures the fieldof view that is viewable to a pair of human eyes. The ET determineswhich direction the user's eyes are centered on. A processor on the EBuses the input from the ET and records an image frame that the user'seyes had seen.

Simplified block diagrams of embodiments of the MVC are shown in FIG. 1and FIG. 2. Referring to FIG. 1, the embodiment of the MVC 100 includesComposite Eye Modules 102 in communication with a first Scene SelectionMicroprocessor module 104, and Eye Tracking Modules 106 in communicationalso with the Scene Selection Microprocessor module 104. The SceneSelection Microprocessor module 104 is in communication with a DSP andMicroprocessor 112. The Physical Sensors 114, Physiological Sensors 108,User Interface module 110, Image Storage module 116, Image Displaymodule 120, Image Transfer module 118, and Image Transmit module 122 arefurther in communication with the DSP and Microprocessor 112.

The Image Transfer module 118 is an interface such as a USB interface tothe DSP and Microprocessor 110, and the Image Transmit module 122 is awired or wireless interface that has the capability of communicatingwith an Ethernet port, a wireless device, a wireless access point and/orone or more wireless networks. In at least one embodiment, the CompositeEye Modules 102 and Eye Tracking Modules 106 may utilize serialcommunication to transfer the image data to the Scene SelectionMicroprocessor module 104. For example, in a particular embodiment,camera chips based on SMIA (Smart Mobile Interface Architecture) or MIPI(Mobile Industry Processor Interface) providing serial data output maybe used. This reduces the number of wires/traces that must run from theframe to the box. Of course, in other embodiments parallel data transferremains an option if the distance between the frame and the box isreduced significantly. For example, the box can be embedded inside adevice attached to head like a headphone.

Referring to FIG. 2, a more detailed embodiment of a MVC 200 includes aframe 214 and a storage/processor box (electronic box) 216 that areconnected through wires. The frame includes Scene Camera Module—Left202, Scene Camera Module—Right 204, Tracking Module—Left 206, TrackingModule—Right 208, and Microphones 210 and 212. The electronic box 216includes the Scene Selection Microprocessor 104, DSP andMicro-controller 112, Attention Sensor 228, Location Sensor 230,Temperature Sensor 232, Acceleration Sensor 234, Wireless Broadcastmodule 218, Storage media 116, Memory device for firmware 220, a PCInterface 222, DRAM Buffer 224, User Interface 110, Status LEDs 226, andLCD or touch screen display 120.

In one embodiment, the Composite Eye Modules consists of Scene Cameraunits 202 and 204 that are in communication with the Scene SelectionMicroprocessor 104, and the Eye Tracking Modules 206 and 208 are incommunication with the Scene Selection Microprocessor 104. The SceneSelection Microprocessor 104, Attention Sensor 228, Location Sensor 230,Temperature Sensor 232, Acceleration Sensor 234, Wireless Broadcastmodule 218, Storage media 116, Memory device for firmware 220, a PCInterface 222, DRAM Buffer 224, User Interface 110, Status LEDs 226, andLCD or touch screen display 120 are all in communications with the DSPand Micro-controller 112.

The MVC 200 includes right and left scene cameras 202 and 204 which areeach coupled to a scene image processor 104. The Scene SelectionMicroprocessor module 104 is further coupled to a microprocessor 112.The MVC 200 further includes a left eye camera 206 and right eye camera208 coupled to the Scene Selection Microprocessor module 104. The eyetracking processor 104 is further coupled to the microprocessor 112.

Each of the scene cameras (202 and 204) capture a portion of the fieldof view of the user of the MVC 200 as image data and provides the imagedata to the scene image processor 204. The scene image processor 204then processes the scene image data and provides the processed imagedata to the microprocessor 112.

The left eye camera 206 provides an image of the left eye of the wearerof the MVC 200 to the Scene Selection Microprocessor module 104 and theright eye camera provides an image of the right eye of the wearer of theMVC 1900 to the Scene Selection Microprocessor module 104. The processor104 then determines the direction to which each of the eyes of thewearer of the MVC 1900 are directed and provides this information to themicroprocessor 112.

An LCD module 120, a storage module 116, and a wireless broadcast module218 are also in communication with the microprocessor 112. The capturedimages are then stored in the storage module 116 and may then be viewedin the LCD module 120 and/or wirelessly broadcasted to wireless networkusing the wireless broadcast module 218.

The MVC 200 further includes status LEDs 226, a PC interface 222,firmware memory 220, a DRAM buffer 224, and a User Interface module 110also in communication with the microprocessor 112. The status LEDs 226may provide information about the status of the MVC to the user. The PCinterface 222 allows communication of the MVC with a personal computeror other computing device. The firmware memory 220 stores the firmwareof the MVC. The User Interface module 110 allows the user to controlvarious aspects and functionalities of the MVC.

The MVC 200 further includes an attention sensor 228, a location sensor230, a temperature sensor 232, and an acceleration sensor 234 incommunication with the microprocessor 112. Each of these sensors may beused to detect an environmental condition or an event and initiaterecording of or storing of a captured scene in the Storage Module 116 bythe MVC 200 in response to the detected condition or event. For example,the attention sensor 228 may detect whether the user of the MVC is atattention and initiate recording of the current scene if such attentionis detected. In a particular embodiment, the attention sensor 228 is abrainwave sensor or measures skin conductivity. The location sensor 230may detect whether the MVC 200 is at a particular location and initiaterecording if such location is detected, or add location information toany recorded video for future search and processing. In a particularembodiment, the location sensor 230 is a GPS sensor. The temperaturesensor 232 may detect that a particular temperature has been detected orwhether a particular change in temperature has been detected andinitiate recording. In at least one embodiment, the acceleration sensor234 may detect that the MVC 200 has experienced acceleration above apredetermined threshold and then initiate recording of the currentscene. In various embodiments, sensor data from one or more sensors maybe stored as metadata associated with the recorded video. In aparticular embodiment, the microprocessor 112 obtains acceleration datafrom the acceleration sensor 234 and inserts the acceleration data asmetadata associated with the recording video. The metadata may then beused during image search and retrieval of the recorded video.Additionally, the acceleration data may be used during image processingto remove motion blur from some image frames recorded by the MVC 200.

It should be understood that in other embodiments, a sensor used todetect any condition or event may be used to initiate recording orcontrol a particular aspect of the MVC 200. In at least one embodiment,one or more of the attention sensor 228, the location sensor 230, thetemperature sensor 232, and the acceleration sensor 234 may be disposedon or within the eyeglasses frame. In still other embodiments, one ormore of the attention sensor 228, the location sensor 230, thetemperature sensor 232, and the acceleration sensor 234 may be disposedupon or within the electronic box.

In various embodiments the Alpha and Beta brain waves of the user orskin resistivity or other cues such as eye movement patterns are usedfor monitoring attention. It is well known that when someone starts topay attention or concentrates, the magnitude of the Alpha waves goesdown from its maximum to almost zero while at the same time theintensity of the Beta waves are increased from zero level to a maximum.In a particular embodiment, brainwave detection includes an antenna thatis placed within the frame and the temple of the eyeglasses that is usedto pick up the Alpha and Beta brain waves of the wearer. The detectedbrain waves are amplified, filtered out from the noise and provided tothe microprocessor 112 within the electronic box. In at least oneembodiment, the MVC uses the brainwave detector to automatically startthe recording process.

In the embodiment illustrated in FIG. 2, the scene cameras 202 and 204,left eye-camera 206, and right eye camera 208 are disposed within or onthe eyeglasses frame and the remaining components are disposed withinthe electronic box. In still other embodiments, one or more additionalcomponents illustrated as being within the electronic box may bedisposed on or within the eyeglasses frame.

In at least one embodiment, location and/or date information may be usedfor tagging captured video and audio information. In variousembodiments, a GPS chip is placed on a main board of the electronic boxto record location information for various video segments. In someembodiments, time information is also created and recorded along withvideo.

In various embodiments, various methods may be used for storing videorecorded by the scene cameras 202 and 204. In at least one embodiment,on-board memory within the electronic box may be used to store capturedvideo. In other embodiments, a removable memory, such as an SD card, maybe interfaced with the electronic box or the eyeglasses frame to recordcaptured video. In still other embodiments, the electronic boxcommunicates wirelessly with one or more networks and the recorded videois stored on one or more network attached storage devices. In variousembodiments, the storage device could be a server within a local areanetwork or a server on the Internet.

In various embodiments, the eyeglasses frame and the electronic boxcommunicate with a wired connection and/or a wireless connection. In anembodiment having wired communication, the eyeglasses frame may receiveelectrical power from the electronic box via one or more wires. In anembodiment having wireless communication, a small battery may be placedwithin the temples portion of the eyeglasses frame. In both cases, wiresmay run through the temples for data communication and power delivery.In an embodiment in which the frame is totally passive, no wire goesfrom the electronic box to the eyeglasses frame. Instead, optical fibersmay serve as the communication means between the eyeglasses frame andthe electronic box.

In various embodiments, the MVC 200 may use wireless transmission totransmit the images or video captured by the scene cameras 202 and 204to a server for live broadcast to select users or for furtherprocessing. With a live broadcast a user can share his or herexperiences with others which may be useful for social networking orreal-time collaborations. In still other various embodiments, the MVC200 can be set to be always on as one mode of operation. In this mode ofoperation, the MVC 200 uses a user adjustable circular buffer thatcovers a predetermined time span, for example a one-minute period. Withthis feature, the user has ample time to capture unexpected moments byinitiating storing of the video within the circular buffer.

As described above, in various embodiments the hardware for mind-viewrecorder may be placed within the eyeglasses frame. This allowsinstalling clear or prescription lenses in the standard lens locationswhich may be important for many people who use corrective lenses.

In various embodiments the Scene Selection Microprocessor module 104 hasa power supply and the associated circuitry on its board. For thisconfiguration, in at least one embodiment it is possible to use a smartphone as the remote storage/processor box 104 as various smart phonesalready have location sensor, Bluetooth, User Interface,DSP/Microprocessor and access to a wireless network and the Internet. Insome embodiments, one or more microphones disposed on the eyeglassesframe 214 can be used for hand-free calling as well. This enables smartphones to record digital still images and videos hands-free. A smartphone can also be used as the electronic box in FIGS. 1 and 2 providedsome modifications are made to the smart phone to accept and control theelectronics that are installed within the eyeglasses frame. In aparticular embodiment, the electronic box is a smart phone with newcapabilities and the frame is a hands-free video/audio input gatheringdevice.

Composite Eyes (CE)

To view the world, human eyes can rotate in two directions: up/down andleft/right. The eyes can also zoom, even though a limited amount.Duplicating this capability within an eyeglasses frame is not easyconsidering constraints such as rotation and tilt speed of the lens, therequired hardware and its power consumption. Rather than using a singlelens that can be tilted in two orthogonal axes, various embodiments usean array of fixed lenses in the viewing optics to cover all the anglesthat an eye covers for a fixed direction of head movement. Thiseliminates the need to rotate or tilt a viewing lens. Also smallerlenses are less expensive and cause less distortions. In fact, invarious embodiments the array of fixed lenses collects all theinformation all the time and because of this feature the eye can betracked in real time. As indicated, the MVC can capture all the detailsthat the user might have or might not have paid attention to. This givesthe user a “second” chance to review the visual images that he/she wasexposed to. In at least one embodiment, the MVC has three displaymodes: 1. Show what eyes see (gazing view); 2. Show gaze view andperipheral view; and 3. Interactively display a portion of the totalfield but chosen by a second user. The third display option is usefulfor remote collaborations or interactions. For example, a nurse might beexamining a patient in a remote area and a doctor in another locationcan choose what to focus on with higher resolution for objects that arewithin the FOV of the nurse. In effect, the eye movements of the doctorchooses scene of interest for him/her. This will be an enabler fortelemedicine.

Fixed-lens cameras are converging systems and as a consequence, theimage size of an object on the detector (camera's retina) becomessmaller as the object moves away from the camera. In human eyes and mostcameras the focal length of the system is changed to get a better viewof the object. It is possible to extract the object distance from stereoimaging of the scene or from tracking the two eyes. Knowing thedistance, the image size can be adjusted similar a human eye. At leastsome embodiments limit the MVC to duplicating the human eye zoom rangeand the frame size of the video that it captures is similar to the frameimages that a human brain receives. Seeing like a human eye makesembodiments of the device suitable to serve as a memory aid for manypeople, especially those with memory disorders. In fact such videorecordings provide subjective visual perception of the user'senvironment for the first time.

Optical Modules within the Frame

FIGS. 3a-3c illustrate an embodiment of the eyeglasses frame for the MVCand the locations of various modules with respect to each other. FIG. 3ashows an embodiment of the MVC frame 300 consisting of the IR-LED 302,Eye Tracking Cameras 306, Viewing Lenses 304 and 308, Lens Holder area318, Eyeglasses Bridge 320, Left and Right Camera Modules 310 and 312,Microphones 316, and Temples 314. In the embodiment illustrated in FIG.3a , the two Right and Left Camera Modules 310 and 312 form thecomposite eye optics 102 of the MVC, and the Tracking Cameras 306 andIR-LED 302 form the Eye Tracking Modules 106.

An MVC has two groups of cameras or lenses. The embodiment illustratedin FIG. 3a shows an eyeglass frame having a left template 314L (or leftarm) coupled to a left eyeglass lens holder area 318L, and a righttemplate 314R (or right arm) coupled to a right eyeglass lens holderarea 318R. The left eyeglass lens holder area 318L is coupled to theright eyeglass lens holder area 318R via an eyeglass bridge 320. SceneCameras can record all or a subset of the user's peripheral view.Tracking Cameras monitor the eyes of the user. In the particularembodiment illustrated in FIG. 3a , a left camera module 306L isdisposed within 318L proximate to the left eyeglass lens holder area318L and a right camera module 306R is disposed within or on the righteyeglass template 318R proximate to the right eyeglass lens holder area318R. A left tracking lens 306L is disposed in or on the left eyeglasslens holder area 318L proximate to the left camera module 310, and aright tracking lens 306R is disposed in or on the right eyeglass lensholder area 318R proximate to the right camera module 312. The leftcamera module 310 and the right camera module 312 further include a leftviewing lens 304 and a right viewing lens 308, respectively. Note thatthe infra-red illuminating LEDs that illuminate the eyes surfaces in oneor more embodiments are marked as 302 in FIG. 3 a.

FIGS. 3b and 3c illustrate embodiments of cameras and infra-red LEDconfigurations for the MVC. FIG. 3b illustrates the front-view of anembodiment of the frame and the embedded Scene Camera Modules 310 and312. FIG. 3c illustrates the back-view of the frame where the IR-LEDs302 and tracking cameras 306 are visible. The IR-LEDs 302 illuminate theeye surface and the eye tracking camera pairs 306 point towards users'eyes for taking images of the eyes. The captured images, which in thepreferred embodiment are in black and white, are used to map out thelocation of the center of the eye pupil of each eye. It is well knownthat the eye pupil absorbs much more infrared light than the tissuesaround it. This makes the pupil the darkest object in captured images.In a particular embodiment, the detectors used for eye tracking have aresolution of 320×240 or less, and the collecting lens may be a pinholelens with visible light filter that transmits infrared and blocksvisible light. The image gathering lens of the said eye tracking cameramay intentionally be set slightly out of focus (defocused) to averageover small features in the image of the eye in order to blur out thesmall features. In such an embodiment, the intentional defocus speeds upthe search for a pupil within the frame.

In the embodiment illustrated in FIG. 3c , the eye monitoring lenses orcameras 306 are embedded in the eyeglasses frame 340 below the right andleft lenses 322 a & 322 b, respectively. The left lens 322 a has fourinfrared LEDs or lensed fibers 302L mounted in or on the circumferenceof the left lens 322 a on the eyeglasses frame 340. The right lens 322 bhas four infrared LEDs or lensed fibers 302R mounted in or on thecircumference of the right lens 322 b on the eyeglasses frame 340.

In various embodiments, the sources of the illumination light areInfrared LEDs. The IR-LEDs can be placed in the inner side of theeyeglasses frame 340 or be housed in the main electronic box 104. In thelater case, lensed optical fibers are used to bring the light to theframe area 214. Lensed optical imaging fibers can also be used totransmit the viewed images of the eye to the electronic box 104 forprocessing of the eye tracking data. In a particular embodiment aninfra-red filter may be used to block the visible ambient light and passthrough the light from the IR-LEDs.

In various embodiments, all of the elements that are needed in theelectronic box are found or can be easily added to smart phones. Hence,in various embodiments it may be preferable to use such a smart phone asthe electronic box 104 because almost everyone is or will be carrying asmart phone. In still other embodiments, phone capability may be addedto the electronic box 104. Permanent prescription or sunglasses lensescan also be installed in the MVC frame 214, as shown in FIG. 3 c.

As it is desired to increase the FOV and at the same time keepresolution of the captured images, there is a need to increase thenumber of monitored pixels per second. This effectively means a biggerpipe is needed to transmit the recorded pixel values from the eyeglassesframe to the electronics box. Using wires make the eyeglasses frameheavier. To address this issue, in at least one embodiment opticalfibers are used to send the signal outputs down to the electronic box.

FIG. 4a illustrates a schematic diagram of an embodiment in which cameraoutputs are transmitted from the eyeglasses frame to the electronic boxvia a fiber Data from Scene Camera Module 402 is received by the Lasermodule 404 and is converted from an electrical signal into an opticalsignal. The modulated laser light 404 is coupled into the optical fiber406 to transmit the light from the frame to the processor 410 in theelectronic box. In a particular embodiment, the transmitter source(laser module 404) is a VCSEL (Vertically Coupled Surface EmittingLaser). Such components are extremely small in size, consume low powerand have very high reliability rates. In addition, a driver may beintegrated with the source to further reduce the space and powerconsumption. State of art power consumption for a driver and a VSCEL at10 GB/s is about 15 mW. A metal coated fiber may be used for electricalpower deliver to the frame.

In FIG. 4b , an embodiment 420 is illustrated in which fused fibertapers 424L, 424M and 424R are utilized to couple lenses outputs oflenses 422L, 422M and 422R into a common CMOS detector 426. As anexample, the fused fiber tapers by Schott can achieve 100 LP/mm ofresolution. An advantage of using a single detector is reducing theelectronics in the eyeglasses arms. By using a larger detector, the sameAnalog to Digital Convertor (ADC) can be used as well as the same imageprocessing unit and reference clock. Consequently, a reduction in powerconsumption may be achieved compared to the case of in which multipledetectors are used.

Eye Tracking Algorithms and Procedures

A number of techniques have been described for eye tracking such asthose described in U.S. Pat. Nos. 5,481,622; 7,736,000; 5,231,674;5,956,125; 7,633,527; 5,892,566; 7,682,026; 7,391,887; 7,391,887;7,259,785; and 7,697,032, all incorporated herein by reference. In allthese techniques, an image processing technique is employed.

An embodiment of an eye tracking algorithm 500 is illustrated in FIG. 5.In step 502, an eye camera, such as left eye camera 306L acquires animage of an eye of the user of the MVC. In at least one embodiment, theimage is black and white. In step 504, the image is converted into abinary image in which all pixels are either black or white. The binaryimage conversion is performed by choosing a predetermined threshold andconverting all pixels at or below the threshold to black, and all pixelsabove the threshold to white. The black and white image includes one ormore “blobs” or black areas indicating possible locations of the pupilof the eye of the user. In step 506, a blob detection algorithm isperformed to detect the blobs within the binary image. In step 508, afiltering algorithm is performed on the blobs to identify the blob thatis the most likely candidate for the location of the pupil. In step 510,it is determined whether the pupil has been found. If the pupil has notbeen found, the procedure continues to step 512. In step 512, thecriterion for converting the image to a binary image is revised and theprocedure returns to step 504 in which the acquired image is convertedto a binary image. The refining or updating includes choosing a newthreshold for binary conversion. For example, if the chosen thresholdresults in too many pixels being identified as black, the threshold canbe raised. If it is determined in step 510 that the pupil has beenfound, the procedure continues to step 514. In step 514, the location ofthe pupil is reported to the microprocessor 104, and the procedurereturns to step 502 in which a new image is acquired. By determining thelocation of the pupil, the direction that the user of MVC is looking canbe determined.

While the techniques illustrated in FIG. 5 can be implemented easily ona desktop or laptop computer, they will typically require too muchprocessing power and battery consumption, and are not suitable formobile devices such as smart phones. The complexity arises from the factthat there are too many candidate blobs (connected regions within theimage frame) that have to be evaluated before the pupil is found.

In various embodiments, that complexity is eliminated or reduced byproper illumination of the eye surface and the setup of the camera. Inat least one embodiment, one or more infrared LEDs are used toilluminate the eye surface of the user of the MVC. In a particularembodiment, the wavelength of the infrared illumination is between 820nm and 950 nm. The one or more infrared LEDs allow obtaining an image ofthe user's eye under varying outdoor and indoor light conditions.Because little or no infrared light is reflected from the pupil, it willbe seen as the darkest area in the image. As a result, an image thatusually has a single blob is obtained. In at least one embodiment, anarray of LEDs is used to illuminate the eye surface and cornersuniformly. This allows very fast detection of the pupil locationcompared to existing schemes which are extremely CPU intensive whichresults in quick battery consumption.

In various embodiments, the illumination level is controlled at startupof the MVC and during calibration. At all times, the IR-LED power levelis maintained significantly below the safe exposure level specified bystandards. In at least one embodiment, the LEDs are modulated to reduceeye exposure by the infrared LEDs by periodically illuminating the eyeof the user during the capturing of the eye image. In a simpleembodiment, a duty cycle of 50% for infrared LED modulation is used forthe periodic illumination of the eye which results in eye exposure beingreduced by half. In still other embodiments, other desired duty cyclesor periods may be used during eye tracking. For example, to use the MVCas a camcorder, the image frame should move gracefully. This means thatthe pupil location is not needed at the frame rate. If the pupil istracked every second, then the eye exposure is further reduced by atleast a factor of 30. It is also possible to alternate tracking the twoeyes which reduces per eye exposure by another factor of two. In aparticular embodiment, only one eye tracking camera is used for eyetracking at a time in order to reduce eye exposure to IR light.

For processing images taken by the eye tracking camera, the requirementof an image memory buffer is eliminated. As pixel values are passed by,the boundary (edges) of the pupil can be determined. If it is assumedthat the pupil center is at the center of the boundary, the location ofthe pupil can be determined. In some embodiments, an option may beprovided to fit an ellipse to the pupil boundary to provide a moreenhanced procedure to track the location of the pupil. Examples of suchprocedure may be found in [1-3].

For image gathering, a black and white camera with a lens that has awide enough FOV to cover the eye region is used in at least oneembodiment. The camera generates gray scale images. For each user, thepupil will move in only a subset of the whole image frame. Duringinitial setup of the MVC, this subset is determined with some addedmargin for error. In all subsequent processing, only that subset isprocessed. In some cameras, the camera can also be programmed so thatonly the desired sub-window is reported as the output image. Uselesspixels are ignored to lower power consumption.

A common step in most image processing routines for eye tracking is toconvert a gray scale into a binary image (black and white) by properchoice of a threshold voltage. An example of a processing an acquiredimage to produce a binary image including a pupil is illustrated inFIGS. 6a-6c . In FIG. 6a , a gray scale image 602 of an eye of the userof the MVC as acquired by an eye-camera is illustrated. FIG. 6billustrates its corresponding histogram 604 and finding an optimum valuefor the threshold voltage 606 in order to determine the location of thepupil. FIG. 6c illustrates the resultant binary image 608 produced byapplying the threshold voltage to the gray scale image 602 in a binaryconversion procedure. With this approach, a single blob may be obtainedin the binary image which is easy to process. FIG. 6c illustrates theresult of an algorithm to locate the center of the pupil which islocated at the intersection of line 610 and line 612. In a particularembodiment, alternating image frames of the eye tracking cameras areused to find the proper value of threshold voltage in order to convert agray scale image into a binary image without the need to buffer theimage data.

FIG. 7 illustrates an embodiment 700 of a simplified and CPU-efficientalgorithm for pupil detection which can be implemented in hardwarewithout image processing routines. This approach relies on the fact thatmulti-IR-LED illumination of eye surface results in a single blob whenthe gray image is converted to binary. The embodiment includes a findthreshold module 706, a convert to binary image module 708, a detectedges 710 (in one dimension as each data row is received), previousthreshold value 702, set new threshold value 704, estimate pupil centerand size 714, validate new pupil 716, minimum and maximum pupil size712, previous pupil location 718, and location and size of new pupil720. A control decision circuit 730 at the input finds the thresholdvalue according to every other frame. The find threshold module 706receives all pixel values and forms a histogram. It furthermore findsthe location of the first minimum in the histogram data. The set newthreshold level 704 receives the new threshold value and compares it tothe previous value before it sets the threshold value level that is usedby the covert to binary image module 708. Set new threshold level 704also resets the flag value so that the next image frame will beprocessed for pupil location. The convert to binary image module 708converts the digitized pixel value to a binary value one pixel at a time(serial processing). If the pixel value is less than threshold, it isset to zero; otherwise to one. The convert to binary image module 708then sends the pixel data, the to the detect edge module 710.

The detect edge module 710 detects the edges of the binary image andoutputs the edge information to the estimate pupil center and size 714.This estimate in fed to the new pupil 716 in which the estimate isquestioned based on the constraints from the minimum and maximum pupilsize 712 and previous location of pupil 718. The final pupil informationis fed to the Scene Tracking Processor 104. The maximum pupil radiusvalue is indicative of the maximum radius that a pupil can be expectedto be in an image, and the minimum pupil radius value is representativeof the minimum pupil radius that a pupil can be expected to be in animage. The max and min values may be used to discard erroneousestimates.

In a particular embodiment, an eye tracking algorithm employs thefollowing steps: a) read-in all the pixel values; b) find their averagevalue and form a histogram in a range between lowest and the averagevalue; c) look for the first minimum after the peak to set a thresholdvoltage; d) use the said threshold to convert the next image into abinary on the fly; e) read-in the image values and record the boundaryof the pupil; f) use the boundary values to find the pupil center; g)use the center and boundary values to find out if new pupil location isacceptable; and h) report the pupil center and the boundaries.

A Simpler Eye Tracking Approach

A second solution is even simpler than the one described in FIG. 7. Thenew approach does not require forming a histogram and finding athreshold, and converting the image into a binary. To achieve these,processing of an M×N image array is reduced into processing of twoone-dimensional arrays of the lengths of M and N. In other words,digital image processing is reduced to digital signal processing. If thecomplexity of the image processing (2D) is of the N×N order, the signalprocessing approach has a complexity of the order N.

Assume the image I has M rows and N columns and form two vectors (dataarrays): Row vector R and Column vector C. R has N elements and C has Melements. The i-th element of C is the minimum value of the i-th row ofthe image I. The j-th element of R is the minimum value of the j-thcolumn of the image. Mathematically R and C can be written as:

C(i)=minimum value of I(i,:), where I(i,:) is the i-th row of image I,and

R(j)=minimum value of I(:,j), where I(:,j) is the j-th column of imageI.

In other words, R is the minimum-hold of the columns and C contains theminimum-hold values of the rows. Minimum-hold circuits can beimplemented in hardware easily.

In a at least one embodiment, an eye tracking algorithm employs thefollowing steps: a) form two minimum-hold row and column vectors out ofthe image data; b) smooth the two vector data using a low pass filter;c) take numerical derivative of each vector; d) find the center andboundaries of the largest and widest valley in each vector; e) use thecenter and boundary values to find out if data is acceptable; and f)report center coordinate as the pupil center and the boundaries asestimate of the size of pupil. In a particular embodiment, the low passfilter is a median filter.

A numerical example of a second algorithm for pupil detection which canbe programmed into camera on a chip is illustrated in FIG. 8. As anillustrative example, a matrix of 6 rows and 8 columns is shown in FIG.8. The minimum values for each row and columns are also listed under Cand R vectors.

This technique is applied to a real eye image that was illuminated withsix IR-LED. FIG. 9 illustrates an image of an eye that is illuminatedwith multi IR-LED. FIG. 10 shows the calculated Row Vector R 1002 forthe image shown in FIG. 9 and FIG. 11 shows the calculated Column VectorC Vector 1102 for the image shown in FIG. 9, respectively, after beingsmoothed by a median filter. In at least one embodiment, a low passfilter may be used. The large dip in each plot corresponds to theboundaries of pupil and the center of each dip is the center of pupil inhorizontal and vertical directions. The following additional arrayprocessing are steps utilized to find the coordinates of the pupilnumerically.

First, the derivative of each array is found numerically to find theboundaries of the pupil in each direction (vertical or horizontal). Thetwo derivative arrays are denoted by R1 and C1. For the image shown inFIG. 9, the calculated derivative of the Row Vector, R1 derivativevector 1210, and calculated derivative of the Column Vector, C1derivative vector 1310 are shown in FIGS. 12 and 13, respectively. InFIG. 12, the two peaks 1202 and 1204 mark the horizontal boundary of thepupil. In FIG. 13, the two peaks 1302 and 1304 mark the verticalboundary of the pupil.

It should be noted that once R1 and C1 are calculated, there is no needfor the data in R and C vectors. Therefore, R and C memory spaces can bereused for storing R1 and C1. In general there may be many small peakson the R1 and C1 curves. These peaks have no useful informationregarding the pupil location. Therefore, it is beneficial to smooth thederivative vectors before identifying the pupil boundary via thefollowing three rules:

1. The signs of the derivatives on the boundaries of pupil are anegative local minimum followed immediately by a positive local maximum.In fact, for proper eye illumination, these minimum and maximum valuesare the global minimum and maximum.

2. The distance between the two boundaries must fall within the maximumand minimum size of pupil. These max and min values can be stored in alook up table for each user. In general, a default max and min valuesfor pupil size can be set.

3. Midpoint of the two boundary points is location of the pupil centerin that direction.

The location of the pupil center 1402 calculated through this method isshown in FIG. 14 as superimposed on the original eye image.

Once the R and C vectors are known, there is a yet another simpleapproach to extract pupil coordinates from the two vectors. FIG. 15illustrates an embodiment of a simplified and CPU-efficient algorithmfor pupil detection which can be implemented in hardware as illustratedin the flowchart 1500. R and C vectors are the two inputs. In step 1506,a histogram for at least R or C is formed. From this histogram, athreshold value is found and is fed to Set New Threshold 1504. This SetNew Threshold unit 1504 compares the new value with the old ones and ifit is valid, it supplies that to 1508 to convert the R and C into binaryvectors. Otherwise, no processing happens until a new threshold isprovided to 1504. The binary R and C vectors are passed to 1510 and inthis block the index of the leading and trailing edges are recorded andfed to 1512. In 1512, midpoints of the two edges are found and reportedto 1514 as pupil location. The Validate New Pupil block uses inputs fromPrevious Pupil Location 1518 and allowed Minimum & Maximum Pupil Size1516 to confirm the new pupil location. This data is reported to NewPupil Location and Size 1520 which in turn feeds that data to the SceneSelection Microprocessor 104.

With a minor change in criteria the above algorithm can be used tolocate the reflection points due to the IR-LEDs within the image.Specifically, rather than looking for the darkest part in the eye image,brightest spots are found through maximum-hold vectors (minimum-hold isused to find the pupil in the eye image). A number of pieces ofinformation can be extracted from the locations of the IR-LEDs. Forexample, the number of illuminating LEDs can determined. In addition,existence of the bright spots due to reflections and the dark spot dueto the pupil indicate that the user is wearing the camera. Given thatthe locations of the LEDs are known within the eyeglasses frame andgiven the average curvature of human eye surface, the locations of theLED images on the eye surface can be used to estimate the distance ofthe frame from the eye surface.

In embodiments of the design described herein, stereo eye tracking isused to independently track each eye. When the eyes focus on a scene,each eye axis makes a different angle with respect to the frame. Withthis information, which is a basis of triangulation technique (knowingthe length of the base and the two angles that have a common side), thelocation of the third corner of the triangle corresponds to the distanceof the user from the scene of interest.

Implementation of Presented Eye Tracking Approaches

Camera chips perform a number of processing tasks on the pixel valuesbefore an image is provided at the output of the device. These tasks arecollectively called “image pipeline” and they include: black clamp, lensdistortion compensation, fault pixel interpolation, white balance, CFA(color filter array) interpolation, gamma correction, color spaceconversion, edge enhancement, auto-exposure and image conversion andcompression.

The three algorithms that were presented for eye tracking require muchless computation resources and memory than the tasks that are currentlyperformed on camera chips. Additionally, for eye tracking, many of thosetasks might be skipped. The conclusion is that the existing firmware oncamera chips can be modified so that all the processing is done on thecamera chip. This results in a power saving and compactness of the MVCcompared to the case where processing occurs outside the camera chip. Inat least one embodiment, the eye tracking cameras are programmed toexecute the eye tracking algorithms described herein directly and reportpupil location to the Scene Camera Microprocessor 104. In a particularembodiment, the eye tracking cameras may be set to down-sample the imagedata before it is transferred to the Scene Camera Microprocessor 104.Furthermore, since the only data that the camera chip needs to send tothe Scene Camera Microprocessor 104 is the pupil coordinate, the samelow speed communication and camera control channel such as I2C can alsoserve as eye tracking data transmission channel.

When the eye images are needed at the electronic box, it is preferred totransmit the image data via a two wire transmission line such as atwisted pair. The data format may be analog (such as NTSC) or serial(such as BT565).

High Resolution Imaging

In the foregoing discussion, embodiments of a dynamic and hands-freeMind-view Video Recorder (MVC) are described. In one implementation ofthat device a pair of eyeglasses' frame is modified to enclose a lensand CCD or CMOS and its associated electronics. To be practical, themind view recorder should be lightweight, easy to wear and have anon-distractive appearance, i.e., be aesthetically pleasing. To achievethis goal, it is desired that the camera module (lens and the detector)be as small as possible. One way to lower the size is to use smallerlenses and detector chips with lower pixel counts. However, thisdegrades the resolution and quality of the video. In the followingdiscussion, embodiments of various techniques and devices are describedto obtain high resolution video using small optical lenses or cameramodules.

These embodiments can be classified into two groups: spatial or temporaltechniques. In both groups, one wide angle camera images the whole sceneof interest. It is this image that needs its resolution to be increased.A second camera is also used to acquire additional detailed informationabout the scene. The images of this second camera are used to enhancethe images of the first camera or are used directly when the user'sgaze-point falls within the FOV of this second camera. In the spatialtechnique method, the second camera takes images of a subset of thewhole FOV with an increased resolution. In the temporal technique, thesecond camera has the same FOV as the first camera but it is samples thescene at the same or higher frame rate.

Approach #1: Spatial Techniques

One way to achieve higher resolution for a fixed lens and image sensoris to use Super Resolution (SR) techniques [4-12]. Bandwidthextrapolation is one of the well-known SR techniques [8-10] whichenforces constraints on the image in space and spatial frequency domainin an iterative manner. The mathematical basis of bandwidthextrapolation has been known for about three decades [8-10] and itrelies on two fundamental theorems described in [8]. These two theoriesstate that [8]:

“Theorem 1. The two-dimensional Fourier transform of a spatially boundedfunction is an analytic function in the frequency plane.

Theorem 2. If an analytic function in the frequency plane is knownexactly in an arbitrarily small (but finite) region of that plane, thenthe entire function can be found (uniquely) by means of analyticcontinuation.”

The key determining factor for successful and practical implementationof SR is to characterize the underlying analytic function even in asmall region but as much as possible. In fact, the resolutionimprovement is limited to the accuracy of the extra information that aregathered about the image. If the function is known exactly in a verysmall area, the image can be recovered precisely. Next a resolutionenhancement implementation based on the stated theorems is presented.

In one design, a wide angle camera takes images of the whole FOV ofinterest (central and peripheral view). This camera is referred to as aLow Resolution (LR) camera. The objective is to increase the resolutionof the images taken by this LR camera. To achieve this, a portion(subset) of the whole FOV is imaged with a second but higher resolutioncamera having a narrower field of view. This High Resolution (HR) imageincreases the information about the scene and is utilized to increasethe resolution of the lower resolution (LR) images. This arrangement isa unique practical and computationally efficient implementation of theabove two theorems. In a particular embodiment, both the wide and narrowangle cameras use the same kind of image sensor.

As a numerical example about the magnitude of resolution enhancements,consider two cameras that have the same image sensors but one have afield of view that is four times larger than the other one (2×horizontal and 2× vertical). Furthermore, assume that each image sensorhas 4 million pixels (2000×2000) and these cameras are used to record720p HD video that requires 1 million pixels per frame. Through pixelbinning, the narrow angle HR camera can capture a subset of the scene atvarious binning levels. For example, for the same binning on both camera(four to one- to convert 4 million pixels into 1 million), theresolution of the narrow angle image is four times higher. Thisadditional information can be further increased by another factor offour (total of 16× resolution increase) if only select 1 millionneighboring pixels on the high resolution camera around any area ofinterest are selected. This is the same as selecting a sub-window on theHR camera without any binning. Of course, this sub-window can move alongthe image sensor area as needed. So from theory for the above example,the resolution of the LR-camera can be increased by a factor of 16. Thisis equivalent to optical zooming for the wide-angle scene. Such opticalzooming cannot be achieved via any point and shoot camcorders.

Any imaging system adds distortion and noise to the real image. Thecommon term for image distortion is blurring and image deblurring is aninverse problem. There are a large number of academic papers thatdescribe how to solve such problems [13-17].

Depending on the Point Spread Function (PSF) of the LR and HR lenses,there are two methods that can be used to enhance the LR image. The PSFof standard lenses is spatial-variant. For such lenses, methods such asthose described in [13-14] can be used. However, with the help of the HRcamera, additional and detailed information about the scene leads tobetter performance with less numerical computations. In the case of MVCcameras, the PSF of the utilized lenses as well as the characteristicsof the image sensors in terms of pixel shape, size, fill factor andnoise are measured and known. With the knowledge of PSF and noise, evensimple deconvolution techniques can do a decent job in improving theresolution of the LR image. Of course, the additional high resolutionimages of portions of the whole FOV which can be used to test theaccuracy of the result.

In contrast to standard camera lenses, wavefront coded lenses [18-21]have a spatially invariant PSF. This allows the use of simplertechniques based on inverse problems to enhance the LR images. In aparticular embodiment, both the wide angle and the narrow angle camerasutilize wavefront coded lenses to extend the depth of focus. Given thatthe PSF's of both camera lenses are space invariant (and can be known ormeasured) and given that both cameras use similar image sensors (samehardware and software), the LR image can be treated as the “blurred”version of the scene (and the HR image). The task then becomes to“deblur” the LR image with the help of HR images that are collected bythe HR camera. It should be noted that the two LR and HR cameras areplaced within the same solid eyeglasses frame and hence the onlydifference between their coordinates is a lateral displacement when thetwo cameras are coplanar. It is also possible that the optical axes ofthe two cameras are not exactly parallel and there is a smallconvergence angle. In various embodiments, image projection andcoordinate transformation is used to make images of the wide and narrowangle cameras into the same Cartesian coordinate. Image projection anddisplacement has been discussed in many computer vision textbooks. Inany event, with the help of image registration techniques [22-24], thetwo images have the same coordinate before they are processed. Invarious embodiments, image registration techniques are used to transferone camera's coordinates to the other camera. Examples of imageregistration techniques include Landweber technique, Block-Matchingtechnique, and Fast Fourier Transform (FFT) technique.

In a particular embodiment, image deconvolution techniques are used toestimate the blurring function of the wide-angle camera with respect tothe narrow-angle high resolution camera. In another embodiment, imagedeblurring techniques are used to estimate the blurring function of thewide-angle camera with respect to the narrow-angle camera. In stillanother embodiment, inverse formulation is used to estimate the blurringfunction of the wide-angle camera with respect to the narrow-anglecamera. In various embodiments, camera characteristics are stored in alook up table and the look up table is used by the image processingtechniques to increase the resolution of the LR image.

FIG. 16 shows the relative field of views captured by the two wide andnarrow angle cameras in which the narrow angle camera captures a highresolution image (q2) of a subset of the wide angle view camera (g). InFIG. 16, the wide angle LR image is denoted by g where g=g1+g2. The truehigh resolution version of the same scene is f where f=f1+f2 and g1 andg2 are the low resolution images of f1 and f2, respectively. The HRcamera records the area denoted by q2 (after image registration andtranslation). Assuming that the composite blurring functions h1 and h2relate f and g, and q2 and f2 through convolution operation (g=h1*f andq2=h2*f2), then, g2=h1*f2.

To increase the resolution of the first camera to the level of thesecond camera, it is assumed q2=f2. With this assumption and via variousinverse problem techniques, h in g2=h*q2 can be found because both g2and q2 are known and measured, and the noise and distortion of bothcameras are known. In the above expressions, the noise terms wereignored to describe the approach.

A special case of the spatial approach occurs when the second camera hasthe same FOV and image sensor as the first camera. Even in this case, ahigher resolution image of the scene is obtained by proper positioningof the second image sensor with respect to the first one. The trick isto make the second camera sample the scene with minimum overlap withpixels of the first camera. FIGS. 17a and 17b show two configurations toachieve higher resolution sampling of the field of view throughorientation diversity (FIG. 17a ) and via orthogonal orientation ofrectangular pixels (FIG. 17b ). For example, the second image sensor maybe rotated by 45 degrees with respect to the first image sensor if thepixels have a square shape (FIG. 17a ). While the two image sensorsstill have similar electronic characteristics, their pixels neveroverlap. In the case of rectangular image pixels, the second imagesensor will be rotated by 90 degrees with respect to the first sensorfor maximum image enhancement. In fact, a good combination is havingrectangular pixels with aspect ratio of 2:1. In this case, one camerasamples the scene in horizontal direction and the other in the verticaldirection. By combining the two images, total image resolution isdoubled compared to either camera. An additional factor of tworesolution improvement can be obtained through iterative bandwidthextrapolation technique.

In FIGS. 17a and 17b , the image sensors 1710 and 1712, respectively,were chosen to have square shapes with dimensions in such a way tocollect all the light from the image gathering lenses. Hence, for agiven lens, the configurations shown in FIG. 17a and FIG. 17b cover themaximum FOV with 100% light gathering efficiency. For the reference, thelight gathering efficiency in a typical consumer grade camcorder with afixed lens is 54%. In others words, about half of the FOV the lenscollects is not used converted to an image.

In the cases described in FIGS. 17a and 17b , the two scene cameras haveidentical wide angle lenses and the image sensors are also identical butwere positioned differently with respect to the scene. An additionalconfiguration is obtained when the two image sensors are positioned inthe same way with respect to the scene however one of the cameras iscolor and the other one is black and white (B&W). Since the B&W cameradoes not use any color filter, it has a higher signal to noise ratio andresolution. Furthermore, it does not suffer from numerical artifacts dueto interpolation that is required to estimate missing color values ateach pixel. By combining the images due to the two cameras, a colorimage is obtained that has a better resolution than the color image dueto the color camera alone.

In various embodiments, the MVC utilizes eye tracking such as thatdescribed herein to choose a subset of the total FOV that the user isgazing at for resolution enhancement. Therefore, the LR camera canoperate without any binning to record a higher resolution of the sceneand only this subset needs to be enhanced. Also, many times the imagesfrom the HR camera (narrower FOV) can be used directly without any needfor image registration or enhancement when the scene of interest is asubset of FOV of the HR camera. In a particular embodiment, the eyetracking apparatus is also embedded within the same eyeglasses framethat houses the video recording cameras. In various embodiments, the eyetracking unit also provides an estimate of the distance of the objectfrom the cameras. In a particular embodiment, the object distance isestimated via triangulation applied to the scene recording cameras. Instill other embodiments, the estimated distance is used to set the zoomlevel of the images to that of a normal human eye.

FIG. 18 illustrates a scene of interest with respect to the field ofview of two cameras in which image data from the high resolution area Nwill be used to increase the resolution of the area denoted by A.Referring to FIG. 18 and its notations, when an area such as 1808outside the FOV of HR camera 1806 is of interest, only a smaller subsetof 1802 such as 1804 can be chosen in conjunction with 1806 forresolution enhancement. This lowers the amount of required computationsignificantly. Additionally, the size of the HR area 1806 is larger than1804. This further reduces the required computations.

For completeness, it is possible to sample the wide angle FOV with morethan one HR camera as shown in FIG. 19a-19c . FIGS. 19a-19c illustratealternative configurations for high resolution sampling of whole fieldof view via more than one camera wherein FIGS. 19a, 19b and 19c five,three and two cameras, respectively, are used to sample the FOV ofinterest. In FIG. 19a , portions of the desired FOV 1902 are sampled byfive HR imagers with FOVs marked by 1904 a, 1904 b, 1904 c, 1904 d and1904 e. In FIG. 19b , only three cameras with FOVs of 1906 a, 1906 b and1906 c were used to sample the total FOV 1902. FIG. 19c illustrates anembodiment with only two cameras to sample the FOV 1902. The two camerasFOV are denoted by 1908 a and 1908 b, respectively.

In FIG. 20, two configurations are presented that utilize only highresolution cameras to cover the desired FOV 2002. FIGS. 20a-20cillustrate an embodiment of a four-camera high resolution cameraconfiguration to cover the whole field of view directly in which inFIGS. 20a and 20b the left and right camera modules FOV and theiroverlap are shown, and in FIG. 20c the net FOV with redundancies areshown. FIG. 20a shows the total FOV 2002 as well as the FOV of the leftcamera modules 2004L and 2006L. FOV of the right camera modules 2004Rand 2006R are shown in FIG. 20b . In FIG. 20c , it is shown how thetotal FOV 2002 is covered by the four cameras.

FIG. 20d illustrates an embodiment of a two high resolution camerasolution to cover the entire field of view of interest. In this case,the total FOV 2002 is covered by two cameras (one camera on each side ofthe frame) with fields of view marked by 2004L and 2004R. In thediscussed configurations in FIGS. 20a-20d , the overlaps between the FOVof the various cameras is intentionally introduced to ease transitionsfrom one camera to any other.

The solutions presented in FIGS. 20a-20d require less post-processingbecause all the cameras are high resolution (HR). Consequently, only onecamera at a time needs to send images to the Scene SelectionMicroprocessor 104. The other one or three cameras go on standby mode toreduce power consumption. Still super resolution techniques can beapplied to the images to further increase their resolution beyond theresolution of the image sensor of the cameras.

Approach #2: Temporal Techniques

There are a large number of academic papers under resolution enhancementand super-resolution in which multi low resolution images are used togenerate a single high resolution image. A common approach is to usemany frames from a low resolution (LR) video. In essence, these variousframes are images of the same scene with random sub-pixel shift withrespect to each other. A high resolution grid is formed and the LRframes are “registered” and placed on it before the high resolutionimage is constructed [4-7, 12].

FIGS. 21a-21d illustrate graphically the concept of achieving superresolution image enhancement via sub-pixel image shift across the imagesensor. Starting in FIG. 21a , images of two points 2102 and 2104 aredetected by the same pixel 2106M. This means that those two pointscannot be seen separately. In FIGS. 21b and 21c , it is shown by lateraldisplacement of the image across the neighboring pixels by a distance ofa half-pixel, the two points can be distinguished from each other. InFIG. 21b , only 2102 object is detected by 2106 pixel whereas in FIG.21c , only the 2104 object is seen by the 2106M pixel. The situationsdepicted in FIGS. 21a-21c can be expressed mathematically using threelinear equations with three unknowns which has a unique solution. Asshown in FIG. 21d , this demonstrates that through the half-pixel shift,the geometrical resolution of the image sensor has been increased by afactor of two, as if the 2106 pixel consisted of two half-sized pixels.This technique is useful because there is a trade-off between reducingthe pixel size and image quality. At the optimum pixel size, by lateraldisplacement of the image over the image sensor, the resolution of theimaging device is increased. Of course, image processing is needed to“register” those additional images and convert them into a single buthigher resolution image.

In a various embodiments, a second camera includes one or more devicesor configurations to shift an image collected by a lens across the imagesensor by a distance less than a dimension of the image sensor pixel. Inparticular embodiments, the distance of the shift may be a length of apixel, a width of a pixel, or a diameter of a pixel.

In the temporal approach, configurations and designs are used in orderto capture a fixed number of sub-pixel-shifted images of the scene pereach frame of the LR camera. For all the reported cases in this section,two wide-angle cameras are used. FIGS. 22a-22c depict embodiments of twoimaging configurations to shift an image across an image sensor in whichFIG. 22a shows a standard configuration for a lens and an imagingdevice, and FIG. 22b and FIG. 22c illustrate the key additional hardwareto achieve image steering. One camera has a standard configuration asshown in FIG. 22a and is denoted as LR camera. This standard camera 2210a consists of two parts: a lens 2202 and an image sensor 2204. For thesecond camera, there are two possible implementations that are shown inFIGS. 22b and 22c to achieve such sub-pixel shift. In FIG. 22b , amovable transparent glass-plate 2206 is introduced between the lens 2202and the image sensor 2204. In FIG. 22c , a micromechanical mirror 2208reflects the light from the lens 2202 onto the image sensor 2204. Byadjusting the amount of the tilt of the either glass-plate 2206 ormirror 2208, the amount of sub-pixel shift across the pixels can becontrolled. In a particular embodiment, the movable or tiltable mirror2208 is a micro-electro-mechanical device.

FIG. 23 illustrates lateral image displacement ranges that can beachieved with a single glass slab that can be positioned at variousangles with respect to the imager. In FIG. 23, the magnitude of lateralimage shift 2306 that can be achieved out of a 1 mm thick glass-platefor various angular rotations 2304 compared to the horizontal case(flat-glass plate is parallel to the image sensor and the lens plane)are shown. The glass-plate may be placed on two PZT elements that aredriven in synch to cause small tilt of the plate. As seen in thedepicted curve 2302, for one degree rotation in either direction (+ or−), the lateral shift can be 5 micrometers. Current imaging sensors havepixel sizes in that range. Of course, by reducing the plate thickness,or the rotation angle, the shift can be reduced as needed. The plate mayonly have three positions (+angle, −angle and zero) and an image istaken at each position.

Image data for only a sub-window of the total FOV can be recorded toreduce power consumption. The images are taken after each glass-plate ormirror movement or in a controlled manner. This requires synchronizationbetween movement of the mirror or the glass-plate and image capturingdevice. The original image and the sub-pixel shifted images (after imageregistration) are combined through processing to create a higherresolution image. FIG. 24a illustrates image pixels in a standard cameraand FIG. 24b shows the virtual pixels that are obtained by diagonaldisplacement of the image across the imager and their locations withrespect to the real pixels. In FIG. 24a , actual image pixels 2410 of astandard camera are shown with open circles while in FIG. 24b theenhanced version of the same image sensor is illustrated. In FIG. 24b ,the newly introduced virtual pixels 2420 are shown with a plus signinside the open circle. As seen in by comparing the two figures, thepixel resolution has been doubled through sub-pixel image shifting.

In MVC design, two scene camera modules are used—one on each side of theeyeglasses frame. One camera will be standard (has no vibrating element)and the other one has a vibrating mirror or plate. Depending on aparticular application, the two cameras can have similar or dissimilarlenses and imagers.

In one embodiment, two cameras that have identical lenses and imagersare deployed. While one camera takes normal pictures, the other camera(sub-pixel shift) takes pictures with any sub-pixel accuracy. Videosfrom the normal camera can be broadcast or shared with no additionalprocessing. However, the second video has to be processed to produce ahigher resolution video. This additional processing step may be done onthe mobile device (at a higher computing cost) or may be done on aserver during live broadcast, or when the video is downloaded to a PC.As in the first approach, not all the pixel values are to be recordedwhen the objective is to follow the gaze-point of a user.

It should be noted that standard SR techniques require extensivecomputations and most of this computation is used to register thevarious low resolution images properly before constructing a highresolution image. The significance of our proposed techniques is thatthe image shift across the sensor array is controlled precisely andconsequently, all the recorded frames are already registered.

In terms of super-resolution practicality, a number of researchers havealready implemented SR on FPGA platforms [25-29]. Various resolutionenhancement algorithms described in this application are simpler thanthose that have already been implemented in FPGA. To preserve batterylife, the resolution enhancement step may be done as a post-processingstep, or be done in real-time on a remote server.

Additional Implementations

One approach in image super resolution is called learning-based orpatch-based super resolution [5, 28-30]. In this approach, a database ofimages that have common features or shapes are used as auxiliary tools.A related area is enhancing a video of a static scene via HR photographs[29]. In our case, the HR camera takes a HR image of the scene at thesame time as the LR image is taken. This means that MVC does not need toform a database or search for patterns in it. From a differentviewpoint, the images of the HR camera form a relevant temporal databasefor image enhancement. This means that the existing pattern-based imageenhancement techniques can be easily applied to the HR and LR images.

Human eyes cannot see well in the dark but there are cameras (IR ornight vision) that can be used for imaging in dark conditions. Inanother embodiment, a two-mode camera is used to enable recording duringday and night. One camera is normal and it records in daylightconditions, and the other camera has infra-red detectors to record indark conditions. Infra-red detectors of many types may be used. Forminimal improvement, existing CMOS detectors are used in conjunctionwith an optical filter that allows only the IR light to get in andblocks white light (blue, green and red). Other IR detectors have muchbetter quantum efficiencies at IR than CMOS detectors. By using suchdetectors, better night images are obtained. In one embodiment, one sideof the eyeglasses frame has day vision and the other has night vision.

In still another embodiment, additional cameras are placed on the sideof the temples of the eyeglasses frame to achieve a total FOV around 270degrees which is much larger than what human eyes can achieve. Theseadditional cameras are always looking sideways. This is useful forvirtual reality applications or for interactive revisit of a scene or anexperience.

It is possible to separate the optical lenses from the imaging detectorby employing a length of fiber imaging devices such as thosemanufactured by Schott. Such fiber currently can achieve a resolution of50 LP/mm. An advantage of using this scheme is to consume much less orno electrical power in the frame area. In another embodiment, theeyeglasses are made totally passive by using such imaging fibers tomonitor the user's eye for eye tracking purposes. Using an imaging fiberto couple the light from a lens to a detector is applicable to variousembodiments.

Existing image sensors have a fixed aspect ratio of 4:3 for StandardDefinition TV and 16:9 for High Definition TV. In the case of MVC, alarger FOV is needed to select a subset of the FOV for SDTV or HDTVvideo. The optimum aspect ratio for MVC application is 1:1 which meansthe detecting area is square.

In various embodiments, the resolution enhancement processing describedherein may by performed by the image cameras, on a computer on a networkafter the images are transmitted to the computer over the network, or ona personal computer after video images are downloaded from the MVC tothe personal computer. In still other embodiments, the high resolutionvideo capturing device is embedded in a wearable device and the camerasare embedded within the frame of eyeglasses.

In various embodiments, the MVC allows a user the following options torecord the video: a) only what the user saw; b) only the scenes that theuser did not pay attention to; or c) record the videos from all scenecameras and have the option to view what was seen or was not seen.

In still other embodiments, the eye tracking signal for scene selectionis provided by a second person who is wearing a similar eye trackingdevice as the user of the MVC and uses his/her eye movement to select ascene within the FOV of the first user.

In still other embodiments, the MVC cameras use a circular buffer tocontinuously record video. As soon as the user decides to save what isin the said buffer, the camera system will continue recording and thebuffered video becomes the beginning part of the recorded clip.

Personal Search Engine

Given the ease of use of a hands-free video recording device asdiscussed herein, each user may generate a huge amount of recordedpersonal video each year. Therefore, it is very important to be able toindex, search, sort, and retrieve the recorded information to make theMVC a more useful device. To do this, personal search engine softwarethat is configured to crawl through the videos and indexes them as soonas they are downloaded to a server or a PC with proper software may beused in a at least one embodiment. For example, the personal searchengine software may employ voice to text technology and Query by Humming(QbH) to create keywords based on the audio information part of thevideo. Also shape and face recognition are used to further index thevideos. MPEG-4 and MPEG-7 standards make it possible to search forobjects in video images. In various embodiments, the search engineindexes the recorded video based on metadata from any of the physical orphysiological sensors of MVC. In a particular embodiment, QbH is used toindex the audio portion of the video. In other embodiments,Voice-to-Text or Face Recognition is used to for indexing the data. Instill another embodiment, MPEG-7 is used to index objects in the video.In some embodiments, the data and the search engine reside on a serverto which the MVC has access through wireless networks or any othercommunication network.

The search engine can be trained by associating names to faces and thiscan be further used to index personal videos or used as a real time toolfor people who forget names and need them instantly (memory patients areusually embarrassed to acknowledge their limitations to people that theymeet). The search engine is also capable of summarizing a video and indoing so individuals can create daily summaries of their lives orcompile their own life stories by editing the summaries. The dailysummaries can also be posted as video logs (vlogs) online. In a at leastone embodiment, the personal search engine software resides on a networkserver which receives recorded information, such as video and audio,from the mind-view video recorder and stores it within one or morenetwork storage devices. The personal search engine software thenindexes the recorded information and makes it searchable for futureaccess by the user of the device or for sharing with others. In aparticular embodiment, the mind-view video recorder communicates withthe server via one or more wireless networks.

Video summaries can be created in different ways. One criterion forsummarizing videos is based on the recorded brainwaves. Each segmentsummary may be chosen based on the intensity of the brainwaves. Parts ofthe video segment where attention is reduced may be ignored and thetotal length of the summary could be subject to a time constraint.Another criterion for summarization could be location change. Aspreviously discussed, the MVC may employ a GPS or utilize wirelesssignals to extract the location information and use it as a tag ormetadata.

Applications of MVC in Healthcare

MVC as a Memory Aid

MVC records not only video and audio, but it has a number of sensorsthat keep track of temperature and location. In some embodiments, theMVC may function as a memory aid used by people suffering from memorydisorders such as dementia and in particular Alzheimer's. The capturedinformation constitutes at least 90% of “short-term memory” information.With the use of the personal search engine, a user can search andretrieve information or reply some experiences to reinforce user'smemory. Additionally, by utilizing face identification and objectrecognition, a person with cognitive impairments can recognize friendsand family members. The tool may display the name, say it aloud orwhisper in the user's ears. At times a memory patient may misplace anobject in an unusual place. The caregivers can use the tool to locatethe object.

MVC for Tele-Medicine

One of the difficulties that telemedicine faces is the lack of highquality and yet flexible video stream that could match the capabilitiesof a physician's eyes. At a doctor's office, a doctor can at will turnhis head and eyes in any direction for examination. For telemedicineapplication, MVC will let the cameras follow user's eyes when the useris describing the situation and symptoms to the doctor. The doctor maywant to examine certain areas within the field of view of the patientclosely. For this case and with patient's permission, MVC will allow thedoctor to control the cameras via the doctor's eyes. Of course in thiscase, the doctor needs to wear his/her own pair of MVC programmed fordoctors or move a mouse on the region of interest to get obtain higherresolution images from that area.

MVC as a Diagnostics Tool

Eye tracking has been used to diagnose children with attention deficit.Doctors can use MVC to study effects of medications on such children. Bywearing the device before and after medications or long term treatments,doctors can get a measurable indication of efficacy of the medicine ortreatment.

MVC as Feedback Tool for Deep Brain Stimulation and Other SimilarDevices

There are a number of diseases such as Essential Tremor, MultipleSclerosis and Parkinson's that lead to involuntary hands movements orshacking. In extreme cases, a technique such as Deep Brain Stimulationis used. In this approach, tiny electrodes are implanted in the brain.The amplitude and frequency of signals are adjusted by a healthprofessional to minimize the shacking. However, it turn out thatpatients require repeated re-adjustment of the device as theirconditions change. This requires a visit to a doctor or a hospital,which is not convenient. Given the shaking frequency is a few Hertz, MVCcan accurately measure the frequency and magnitude of the shakes. Thisinformation can in return be used as feedback for the device todynamically adjust itself.

MVC as Hands-Free Camcorder and Interface Device for People withDisabilities

Video recording application of MVC is obvious. In addition to videorecording, they could also use the eye tracking interface to do othertasks such to control a wheelchair, or turn on and off a TV or anotherdevice.

MVC for Relapse Prevention

In particular embodiments, the MVC may be used for real-time monitoringof subjects such as to achieve behavioral change in a subject. Relapseis a process and not an instant event. Individuals that go throughrehabilitations need proper follow up monitoring to avoid relapses.Through MVC, face and pattern recognition and the search engine,individuals can be warned to stay away a select group of others andplaces. The individuals can also be randomly monitored for adherence tothe program guidelines.

MVC for Patient-Care

Patients and nurses can benefit from using MVC. A nurse can document howhe/she treated a patient and when and what medications were used. Thisis especially true in case of terminally ill people whose relativescannot be with them all the time to monitor their loved ones as theyreceive care.

MVC as a Tool for Clinical Trials

Clinical trials are very expensive and time consuming procedures.Participants are required to record or communicate their conditions,side-effects or etc. Many times, people delay taking such note becauseit is not convenient or possible. In the process, people may forgettotally or provide an estimate at a later time. In case ofcomplications, clinical trial organizers can review the chain of eventsto better identify the cause of adverse effects.

MVC as a Hands-Free Camcorder for People Who Cannot Use their Hands

People with essential tremor cannot use the existing hand-held videorecorders because their hands shake involuntarily as they want to usetheir hands. This results in blurry images and videos which are hard towatch. With MVC, anyone who cannot use their hands, for any reason,could use their eyes to record a high quality video of the scenes ofinterest.

MVC for Dentist and Hygienists

By wearing an MVC, such professionals can share what they see (as theyperform a procedure) with their patients by displaying what they see ona TV monitor that is usually in the room.

Applications of MVC in Media and Entertainment

MVC for Social Networking

In case of Twitter, users exchange short text messages with each other.With MVC connected to a wireless network directly or using connection ofa smart-phone, users can share their visual experiences in real-time ormay post them online for friends and “followers” to view.

MVC for Extreme Sports

Currently, athletes are seen through TV cameras. With MVC, additionalangles are recorded from the athletes' perspective.

MVC to Capture Unexpected Moments

Everyone has un-expected moments that wish could have been captured. Aroad accident, a child's first step, a burglary in action and many otherunexpected moments can be captured with MVC. This is done throughcontinuous recording to MVC's circular buffer.

MVC as a Tool for Citizen Reporters

Citizen reporters with MVC can be their own cameraperson and reporter.This reduces their cost.

MVC as a Tool for Real Estate Agents

Real estate agents can use MVC to send an overview of a new propertythat they have been informed of to a prospective buyer. Many times,buyers need particular information that is not included in a virtualtour or in the advertisement sheet of the house. There are also housesthat have no virtual tour. An agent who has an MVC can capture theinformation and send it to house hunters. House hunters can also benefitfrom MVC. After seeing more than a few houses, people start to confuseone house for another. In such cases, a visual record of what was seencan save the times of the agents and house hunters by eliminating theneed for multiple visits.

MVC as a Tool for Tele-Presence

For a true tele-presence experience, two camera views are needed: onecamera will look at the user and his/her surrounding, and the othercamera captures what the user sees. A video conferencing camerafunctions as the camera and MVC as the second.

MVC for Making Documentaries and Movies

MVC allows a director to shoot a scene exactly how he/she wants by usinghis own eyes. Actors and Actresses can also record a scene from theirown viewpoints. Such video clips can be edited to make a movie.

MVC as a Tool for Referees for Instant Replies or to Share What they Saw

TV camcorders cannot cover all angles. It will be nice to showspectators what a referee saw before he/she made a call.

MVC as a Camcorder for Animals and Pets

MVC optimized for pets and animals will allow a user to see what theypay attention to.

Applications of MVC in Education

MVC as a Tool for Preparing do it Yourself Instructions

Many products require do-it-yourself assembly. However, the writteninstructions are hard to follow for many people. Preparing aprofessional video of instructions is costly too. However, themanufacturer can use an MVC to show how the product is assembled step bystep.

MVC as a Teaching Tool at Schools

Children have different reading abilities. As a child starts to read anew book, there are words that he/she does not know how to pronounce ortheir meaning. MVC can be an always available resource to schoolchildren. From the analysis of eye movements, it is possible to predictif the reader needs help. MVC can also work as an interface byinterpreting hand gestures. For example, a finger pointed at a word maysignify more information about the word. From a dictionary databasepronunciations or meanings can be provided. The user may also “cut andpaste” a sentence or a paragraph from the book via finger gestures. Inthe same manner as reading application, MVC can be used for matheducation.

MVC as a Feedback Tool for Coaches

Many times players do not perform a task as a coach directs. Forexample, a swimming coach may suggest a particular stroke technique.Capturing and replaying a performance immediately is an invaluableteaching tool.

MVC as an Interface Device Between Man and Machine

When using computers, occasionally there is a need to move the cursor toa new location on the screen. In some embodiments, the eye trackingeye-wear is used as an interface device between man an machine. Insteadof grabbing the mouse and moving it to the new location, eye or handmovements can be used to keep the curser close or at the gaze-point.

In a particular embodiment, the machine is a tunable lens for visionaccommodation. In still other embodiments, the machine may be a personalcomputer, a mobile phone, a home appliance such as a television set, anautomobile, an electronic book, binoculars, or a wheelchair.

MVC as a Feedback Tool and Quality Control for People Working onAssembly Line

When people are observed from close proximity, they may become nervous.There are also times needs to transfer “to-do-it-yourself” instructionsaccurately. In both cases, MVC can be used to monitor someone orcommunicate how a skilled person performs a task. A supervisor sittingin an office can randomly check various stations remotely.

MVC as a Fast Documentation Tool

With MVC one can record visual information at ease while addingcommentary. After severe traffic accidents, a detailed report needs tobe prepared with pictures and information about locations of variousobjects in the scene. These are time consuming activities and result inkeeping roads closed for longer periods of time. MVC can record what anofficer sees and use pattern recognition and triangulation to measureand report various objects and their locations. Also police officers whostop a car or search a property can document their search and findings.

MVC as a Time Management Tool

MVC equipped with pattern recognition and personal search engine can beused to provide feedback on how a user spends his/her time. For example,3 hours in front of TV, 10 visits to refrigerator, 2 hours in front ofcomputer, 90 minutes driving and etc.

MVC for Weight Control

Many weight loss control programs suggest participants to keep a dailylog of what they eat. This is a difficult task for most people on thoseprograms. With MVC and its search engine, an accurate record of whatpeople eat, how many times they eat and etc can be complied. Suchrecords can be used by weight loss counselors to propose propercorrective actions.

Applications of MVC in Safety and Security

MVC for Identifying Missing or Wanted People

MVC follows user's eyes and has wireless access to transmit its imagesto a remote site or server. A security officer equipped with MVC canwalk in a public place such as a mall or in an airport while looking atvarious faces. Images of faces are taken and transferred to a remoteserver for possible match with wanted individuals who might be missingpeople or wanted criminals. Face recognition can be done in seconds andif a match is found, the officer can be informed to take proper action.MVC for Personal Safety and SecurityLone individuals concerned about their personal safety can choose to bemonitored live by a relative, friend or a security service for anyduration of time.MVC for Command and Control Training

Many times trainees go through staged training to learn a task or toshow mastery of the required skills. To provide effective feedback tosuch participants, knowledge of what they saw and what they missed iscrucial. MVC provides an exact copy of what they saw and what they didnot pay attention to. Hence, this video can be used for fact-basedtraining feedback.

MVC as a Wearable Speed Detector

In particular embodiments, the MVC is used as a visual detector tomeasure speed of cars. In such embodiments, the MVC can record in 3D.Stereo imaging can be used to estimate the distance of one or manyobjects from the cameras, for example through triangulation techniques.Existing radar detectors that police officers use are heavy and have tobe aimed accurately at a car to measure its speed. Via the MVC's stereoimaging, a car's speed can be measured and the recorded video can alsobe used as unbiased evidence. In at least one embodiment, eye trackingdata from eye tracking data from cameras are used to calculate thedistance of the wearer of the MVC from a point of interest.

MVC for Multi-Operator Tasks

In hazardous environments, the amount of time that an operator may spendmay be limited due to health risks. In such situation, the next operatorcan watch through the current one to know what exactly has been donebefore he/she continues the work. The same thing can be achieved in amedical emergency. A doctor at a hospital can watch through aparamedic's eyes to understand the situation better and prepare for thepatient's arrival.

MVC for Compliance by Court Orders

Many times individuals are required by law to stay away from certainacts, places or individuals. For example, pedophiles must abstain fromcertain activities while they are on parole. A remote monitoring stationcan simultaneously check on such individuals activities provided theyare wearing their MVC. An eye recognition program can ensure MVC is wornby the right individual.

MVC as a Tool to Broadcast Traffic Conditions or Accidents by Drivers

Drivers can use MVC to broadcast road condition or report details of anaccident to traffic or accident centers.

MVC as a Tool for Teenagers and Elderly Drivers

MVC will see and understand the road signs such as speed limits, stopsigns and traffic lights. It has its own speedometer (from how fastobjects move in a 3D scene). Once worn, it could warn the drivers aboutapproaching a red light, a stop sign, green light turning yellow, speedlimit on the road, getting out of your lane and etc. It is possible thatteenagers will be required by the law to wear this device after oneoffense. For elder drivers with low-vision, MVC could be a driving aidtool.

MVC for Remote Assistance and Collaborations

In particular embodiments, the MVC is provided with direct wirelessaccess to facilitate real-time collaborations. Through the MVC, one cansee through someone else's eyes. This makes the MVC a good tele-presencetool for remote assistance. For example, one can help someone else to doa task as if watching over the person's shoulder. In particularembodiments, the real-time collaboration is for field service andtroubleshooting. In still other embodiments, the real-time collaborationis used for command and control for public safety and security. In otherembodiments the real-time collaboration is used to enabletele-diagnostics or complement video conferencing for tele-presence.

MVC in Emergencies

People panic when an emergency arises. Some can't talk but their eyescan still see. By live broadcasting of what a user sees (or beingactivated by 911), help and rescue teams can learn much more about thecase and become better prepared to take proper actions.

There are other emergencies that MVC can be used to coordinate effortsor provide more info to all involved. For example, when police needs toenter a house, the officers in the front line can use MVC to show whatthey see to the support officers. Similar information exchange canhappen at a fire scene where multiple firefighters view the scene fromdifferent angles. An operation command center can watch all the feedssimultaneously to decide on the best course of action. An officerstopping a driver can immediately take a video clip of his/herinteraction. There have been cases where a police officer is shot deadwhen stopping a driver or running into other such encounters.

MVC in a Battle-Zone

Soldiers can share what they see in the front line with the supportteam. Alternatively, the support team may advise the soldiers on thesituation. A remote doctor may guide a soldier to provide help for selfor others.

MVC as a Tool in Public Safety

In case of an accident, responders come at different times. Through MVCand its live broadcast capabilities, the first responder can providelive video of the scene while at the same he/she is focusing on his/herown duties. Those who survey the accident scene can also use MVC todocument locations of various objects on the scene.

It will be appreciated by those skilled in the art having the benefit ofthis disclosure that this METHOD AND APPARATUS FOR A COMPACT AND HIGHRESOLUTION MIND-VIEW COMMUNICATOR provides a video recorded that isintegrated with eyeglasses to record a scene that is being viewed by auser. It should be understood that the drawings and detailed descriptionherein are to be regarded in an illustrative rather than a restrictivemanner, and are not intended to be limiting to the particular forms andexamples disclosed. On the contrary, included are any furthermodifications, changes, rearrangements, substitutions, alternatives,design choices, and embodiments apparent to those of ordinary skill inthe art, without departing from the spirit and scope hereof, as definedby the following claims. Thus, it is intended that the following claimsbe interpreted to embrace all such further modifications, changes,rearrangements, substitutions, alternatives, design choices, andembodiments.

APPENDIX I References

-   [1] “A Novel Method of Video-Based Pupil Tracking”, Proceedings of    the 2009 IEEE International Conference on Systems, Man and    Cybernetics, San Antonio, Tex., USA—pp. 1255-1262, October 2009-   [2] D. Li, D. Winfield and D. Parkhurst, “Starburst: A Hybrid    algorithm for video based eye tracking combining feature-based and    model-based approaches”, Iowa State University, Ames, Iowa-   [3] “A High Speed Eye Tracking System with Robust Pupil Center    Estimation Algorithm”, Proceedings of the 29th Annual International    Conference of the IEEE EMBS, Kyon, France, pp. 3331-3334, August    2007-   [4] P. Milanfar, Super-Resolution Imaging, CRC Press (2011)-   [5] S. Chaudhuri, Super-Resolution Imaging, Kluwer Academic    Publishers (2001)-   [6] V. Bannore, Iterative-Interpolation Super-Resolution Image    Reconstruction, Springer (2009)-   [7] B. K. Gunturk, “Super-resolution imaging”, in Computational    Photography Methods and Applications, by R. Lukac, CRC Press, 2010-   [8] J. Goodman, Introduction to Fourier Optics, 2^(nd) edition,    160-165, McGraw-Hill, 1988-   [9] R. W. Gerchberg, “Super-resolution through energy reduction”,    Optica Acta, Vol. 21, No. 9, pp. 709-720, (1974)-   [10] A. Papoulis, “A new algorithm in spectral analysis and    band-limited extrapolation”, IEEE Transactions on Circuits and    Systems, Vol. 22, No. 9, pp. 735-742, (1975)-   [11] E. Gur and Z. Zalevsky, “Single-image digital super-resolution    a revised Gerchberg-Papoulis algorithm”, Internation Journal of    Computer Science, 34:2 (2007)-   [12] D. Keren, S. Peleg, R. Brada, “Image sequence enhancement using    sub-pixel displacements”, Proceeding of IEEE Computer Society    Conference on Computer Vision and Pattern Recognition (CVPR '88),    pp. 742-746, (1988)-   [13] J. Bardsley, S. Jefferies, J. Nagy, and R. Plemmons, “Blind    iterative restoration of images with spatially-varying blur”, Optics    Express, pp. 1767-1782, (2006)-   [14] T. R. Lauer, “Deconvolution with spatially-varying PSF”, Optics    Express, pp. 1767-1782, (2006)-   [15] P. C. Hansen, J. G. Nagy, D. P. O'Leary, Deblurring Images:    matrices, Spectra and Filtering, SIAM (2006)-   [16] S. H. Lim, A. Silverstein “Estimation and removal of motion    blur by capturing two images with different exposures”, HP    Laboratories, www.hpl.hp.com/techreports/2008/HPL-2008-170.pdf,    (2008)-   [17] S.-W Jung and S.-J. Ko, “Image deblurring using multi-exposed    images” in Computational Photography Methods and Applications, by R.    Lukac, CRC Press, 2010.-   [18] K. Kubala, E. R. Dowski, and W. T. Cathey, “Reducing complexity    in computational imaging systems,” Opt-Express 11, 2102-2108 (2003)-   [19] E. R. Dowski, and W. T. Cathey, “Extended depth of field    through wavefront coding,” Applied Optics 34, 1859-1866 (1995)-   [20] W. T. Cathey, E. R. Dowski, “New paradigm for imaging systems,”    Applied Optics 41, 6080-6092 (2002)-   [21] S.-H. Lee, N.-C. Park, and Y.-P. Park, “Breaking diffraction    limit of a small f-number compact camera using wavefront coding,”    Optics Express 16, 13569-13578 (2008)-   [22] L. G. Brown, “A survey of image registration techniques”, J.    ACM Computing Surveys, Vol 24, 4 Dec. 1992-   [23] B. Zitova, J. Flusser, “Image registration methods: a survey”,    Image and Vision Computing, 997-1000, 21 (2003)-   [24] P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency    domain approach to registration of aliased images with applications    to super-resolution”, EURASIP Journal on Applied Signal Processing,    pp. 1-14, (2006)-   [25] M. E. Angelopoulou, C.-S. Bouganis, P. YK. Cheung, and G. A.    Constantinides, “FPGA-based real-time super-resolution on an    adaptive image sensor”, Proc. Applied Reconfigurable Computing 2008,    pp. 125-136 (2008)-   [26] Maria Angelopoulou, Christos-Savvas Bouganis, Peter Y. K.    Cheung, and George A. Constantinides “Robust Real-Time    Super-Resolution on FPGA and an Application to Video Enhancement”,    ACM Transactions on Reconfigurable Technology and Systems (2009)-   [27] O. Bowen, C-S Bougani, “Real-time image super-resolution using    an FPGA”, Field Programmable Logic and Applications, (2008)-   [28] W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based    super-resolution”, IEEE Computer Graphics and Applications, pp.    56-65 (2002)-   [29] P. Bhat, L. Zitnick, N. Snavely, A. Agarwala, M. Agrawala, B.    Curless, M. Cohen, S. Kang, “Using photographs to enhance videos of    a static scene”, Eurographics Symposium on Rendering (EGSR) (2007)-   [30] U.S. Pat. No. 7,715,658 B2, May 11, 2010 Apparatus and Method    for Super-Resolution Enhancement Processing.

What is claimed is:
 1. An imaging method for a multi-camera device,comprising: capturing a first image of a scene with a first camera, thefirst camera having a first field of view, and the first image having afirst image resolution, capturing a second image with a second camera,the second camera having a second field of view, and the second imagehaving a second image resolution, wherein the second image resolution ishigher than the first image resolution and the second field of view issmaller than the first field of view, and wherein the second imagecorresponding to a subset of the scene that is within the first field ofview, receiving the first image and the second image by at least oneprocessor, the at least one processor configured for: registering asubset of the first image to a subset of the second image, calculating,based on at least one of the first image and the second image, thedistance of at least one object in the scene from the multi-cameradevice, generating, based at least upon the calculated distance of theat least one object in the scene from the multi-camera device and atleast one of the received images, an output image corresponding to aportion of the scene that is within the field of view of the firstcamera, wherein the output image having at least two image regions, aninner image region and an outer image region, the outer image regionsurrounding at least partially the inner image region, and at least asubset of the inner image region includes a portion of the at least oneobject, wherein the image resolution of the inner image region is higherthan the image resolution of the first image and the image resolution ofthe outer image region is lower than the image resolution of the firstimage, and saving the output image, whereby the inner image region ofthe output image appears to a human eye as sharp and clear and the outerimage region of the output image appears to the human eye as relativelyblurry.
 2. The imaging method of claim 1 further including reducing theresolution of at least a subset of one of the received images via animage processing technique.
 3. The imaging method of claim 1 wherein thefirst and the second cameras are color cameras and generate colorimages.
 4. The imaging method of claim 1 wherein the image resolution ofthe inner image region of the output image is the same as the secondimage resolution.
 5. The imaging method of claim 1 wherein the outputimage corresponding to the subset of the scene that is within the secondfield of view.
 6. The imaging method of claim 1 further includingmonitoring a movement of at least one of the cameras via at least oneaccelerometer.
 7. The imaging method of claim 1 further includingcapturing the location information of the multi-camera device via a GPS.8. An imaging apparatus comprising: a first camera having a first fieldof view for capturing a first image of a scene, wherein the first imagehaving a first image resolution, a second camera having a second fieldof view for capturing a second image, wherein the second image having asecond image resolution and the second image resolution is higher thanthe first image resolution and the second field of view is smaller thanthe first field of view, and wherein the second image corresponding to asubset of the scene that is within the field of view of the firstcamera, at least one processor, in communication with the first cameraand the second camera, the at least one processor configured for:receiving the first image and the second image, registering at least asubset of the first image to a subset of the second image, calculating,based on at least one of the first image and the second image, thedistance of at least one object in the scene from the imaging apparatus,generating, based at least upon the calculated distance of the at leastone object in the scene from the imaging apparatus, an output imagecorresponding to a portion of the scene that is within the field of viewof the first camera, wherein the output image having at least two imageregions, an inner image region and an outer image region, the innerimage region surrounded at least partially by the outer image region,wherein the image resolution of the inner image region is higher thanthe image resolution of the first image and the image resolution of theouter image region is lower than the image resolution of the firstimage, and saving the output image, whereby the inner image region ofthe output image appears to a human eye as sharp and clear and the outerimage region of the output image appears to the human eye as relativelyblurry.
 9. The imaging apparatus of claim 8 wherein the first camera isa color camera and includes a color filter array.
 10. The imagingapparatus of claim 8 wherein the first image and the second image havethe same number of pixels.
 11. The imaging apparatus of claim 8 whereinthe second camera has a color filter array to generate color images. 12.The imaging apparatus of claim 8 wherein the second camera and the firstcamera have two different types of color filter arrays.
 13. The imagingapparatus of claim 8 wherein the at least one processer is furtherconfigured for reducing the image resolution of at least a portion ofone of the received images.
 14. The imaging apparatus of claim 8 furtherincluding a GPS for receiving the location of the imaging apparatus andcommunicating the location information to the at least one processor.15. The imaging apparatus of claim 8 wherein the imaging apparatus isdisposed on an eyeglass frame.
 16. The imaging apparatus of claim 8wherein the at least one processor further configured for initiating animage projection procedure.
 17. The imaging apparatus of claim 8 furtherincluding at least one accelerometer to monitor a movement of at leastone of the cameras.
 18. The imaging apparatus of claim 8 furtherincluding at least one microphone for converting an audio signal into anelectrical signal.
 19. The imaging apparatus of claim 8 furtherincluding at least one speaker for converting an electrical signal intoan audio signal.
 20. An imaging apparatus comprising: a first camerahaving a first field of view and a first color filter array forcapturing a first image of a scene, wherein the first image having afirst image resolution, a second camera having a second field of viewand a second color filter array for capturing a second image, whereinthe second image having a second image resolution and the second imageresolution is higher than the first image resolution, and wherein thesecond image corresponding to a subset of the scene that is within thefield of view of the first camera, at least one processor, incommunication with the first camera and the second camera, the at leastone processor configured to: receive the first image and the secondimage, register at least a subset of the first image to a subset of thesecond image, execute an object recognition procedure to detect anobject in at least one of the received images, calculate, based at leaston one of the first scene image and the second scene image, the distanceof the detected object from the imaging apparatus, generate, based onthe calculated distance of the detected object from the imagingapparatus and the at least one of the received images, an output imagecorresponding to a portion of the scene that is within the field of viewof the first camera, wherein the output image has at least two imageregions, an inner image region and an outer image region, the innerimage region surrounded at least partially by the outer image region,wherein the image resolution of the inner image region is higher thanthe image resolution of the first image and the image resolution of theouter image region is lower than the image resolution of the firstimage, and, save the output scene image, whereby the inner image regionof the output image appears to a human eye as sharp and clear and theouter image region of the output image appears to the human eye asrelatively blurry.